Image processing apparatus for character recognition, control method of the same, storage medium and image processing system

ABSTRACT

An image processing apparatus acquires a plurality of captured images of characters captured in time series, each of the characters including a plurality of segments, recognize the characters captured for each of the plurality of captured images, and determine which one of the characters recognized from each of the plurality of captured images is to be output. The image processing apparatus determines, in accordance with a change aspect in time series of the characters recognized from each of the plurality of captured images, which one of the characters recognized from each of the plurality of captured images is to be output.

BACKGROUND OF THE INVENTION Field of the Invention

The present invention relates to an image processing apparatus forcharacter recognition, a control method of the same, a storage medium,and an image processing system.

Description of the Related Art

Conventionally, there is a known technology in which physical quantitiessuch as temperature, pressure, rotation speed, and distance are measuredby using a predetermined sensor, and a display such as a seven-segmentdisplay displays a measurement result measured by the sensor.

Additionally, there are instances where such a display is dedicated to afunction of displaying a measurement result, and in the case of usingsuch a display, there are instances where recording and saving of ameasurement result are performed by using an image capturing apparatusseparately provided. That is, there is a known technology in whichindication of a display is first captured by an image capturingapparatus and subsequently character recognition processing is performedon a captured image and a measurement result is output or recorded(Japanese Patent Laid-Open No. 2015-197851).

Incidentally, in a case where a character displayed as a plurality ofsegments is captured by an image capturing apparatus, when indication ofa display changes during exposure, there are instances where indicationobtained before the change and indication obtained after the changeoverlap with each other and are captured. For instance, in a case whereindication of a number changes from “4” to “5” during exposure and theindication of these numbers overlap with each other, the indication isrelatively harmless as long as the indication is captured as indicationof a non-existent number, but there are instances where the indicationis captured as indication of a different number (for instance, “9”).

According to the technology proposed in Japanese Patent Laid-Open No.2015-197851, in a case where indication of the display is updated andoverlapping images are captured during exposure by the image capturingapparatus, there are instances where an erroneous character may beoutput from the captured images as a measurement result.

SUMMARY OF THE INVENTION

The present disclosure has been made in consideration of theaforementioned issues, and realizes technology in which even in a casewhere an erroneous character is recognized due to a change of indicationof a display, a finally accurate recognition result can be output.

In order to solve the aforementioned problems, one aspect of the presentdisclosure provides an image processing apparatus comprising: one ormore processors; and a memory storing instructions which, when theinstructions are executed by the one or more processors, cause the imageprocessing apparatus to function as: a recognition unit configured toacquire a plurality of captured images of characters captured in timeseries, each of the characters including a plurality of segments, andrecognize the characters captured for each of the plurality of capturedimages; and a determination unit configured to determine which one ofthe characters recognized from each of the plurality of captured imagesis to be output, wherein the determination unit determines, inaccordance with a change aspect in time series of the charactersrecognized from each of the plurality of captured images, which one ofthe characters recognized from each of the plurality of captured imagesis to be output.

Another aspect of the present disclosure provides, an image processingsystem comprising: an image capturing unit configured to generate aplurality of captured images of characters captured in time series, eachof the characters including a plurality of segments; and an imageprocessing apparatus comprising: one or more processors; and a memorystoring instructions which, when the instructions are executed by theone or more processors, cause the image processing apparatus to functionas: a recognition unit configured to acquire a plurality of capturedimages of characters captured in time series, each of the charactersincluding a plurality of segments, and recognize the characters capturedfor each of the plurality of captured images: and a determination unitconfigured to determine which one of the characters recognized from eachof the plurality of captured images is to be output, wherein thedetermination unit determines, in accordance with a change aspect intime series of the characters recognized from each of the plurality ofcaptured images, which one of the characters recognized from each of theplurality of captured images is to be output.

Still another aspect of the present disclosure provides, a controlmethod of an image processing apparatus, the control method comprising:acquiring a plurality of captured images of characters captured in timeseries, each of the characters including a plurality of segments, andrecognizing the characters captured for each of the plurality ofcaptured images; and determining which one of the characters recognizedfrom each of the plurality of captured images is to be output, whereinthe determining includes determining, in accordance with a change aspectin time series of the characters recognized from each of the pluralityof captured images, which one of the characters recognized from each ofthe plurality of captured images is to be output.

Yet another aspect of the present disclosure provides, a non-transitorycomputer-readable storage medium comprising instructions for performingthe control method of an image processing apparatus, the control methodcomprising: acquiring a plurality of captured images of characterscaptured in time series, each of the characters including a plurality ofsegments, and recognizing the characters captured for each of theplurality of captured images; and determining which one of thecharacters recognized from each of the plurality of captured images isto be output, wherein the determining includes determining, inaccordance with a change aspect in time series of the charactersrecognized from each of the plurality of captured images, which one ofthe characters recognized from each of the plurality of captured imagesis to be output.

According to the present invention, even in a case where an erroneouscharacter is recognized due to a change of indication of a display, afinally accurate recognition result can be output.

Further features of the present invention will become apparent from thefollowing description of exemplary embodiments (with reference to theattached drawings).

BRIEF DESCRIPTION OF THE DRAWINGS

The accompanying drawings, which are incorporated in and constitute apart of the specification, illustrate embodiments of the invention, andtogether with the description, serve to explain the principles of theinvention.

FIG. 1 is a block diagram illustrating an example of an overallconfiguration of an image processing system and a functionalconfiguration example of an image processing apparatus according to afirst embodiment.

FIG. 2 is a view illustrating indication of a segment display accordingto the first embodiment.

FIG. 3 is a view illustrating a configuration in which a single digit ofthe segment display according to the first embodiment is displayed.

FIG. 4 is a view illustrating a table for number recognition in lightingand non-lighting patterns of the segment display according to the firstembodiment.

FIG. 5 is a view illustrating a relation between a change of the segmentdisplay according to the first embodiment and a captured image capturedby an image capturing apparatus.

FIG. 6 is a flowchart illustrating a sequence of operations of outputdetermination processing of the image processing apparatus according tothe first embodiment.

FIGS. 7A to 7D illustrate results of image capture with respect to achange of a number of the segment display according to the firstembodiment.

FIG. 8 is a flowchart illustrating a sequence of operations of outputdetermination processing of an image processing apparatus according to asecond embodiment.

DESCRIPTION OF THE EMBODIMENTS First Embodiment

Hereinafter, embodiments will be described in detail with reference tothe attached drawings. Note, the following embodiments are not intendedto limit the scope of the claimed invention. Multiple features aredescribed in the embodiments, but limitation is not made to an inventionthat requires all such features, and multiple such features may becombined as appropriate. Furthermore, in the attached drawings, the samereference numerals are given to the same or similar configurations, andredundant description thereof is omitted.

Although in the following description, processing of recognizing anumber displayed by using a plurality of segments will be describedusing the present embodiment as an example, character recognitionprocessing in which other characters displayed using a plurality ofsegments are recognized, is also applicable. Additionally, in thefollowing description, an example in which an information processingserver is used as a computer that can perform number recognitionprocessing (character recognition processing) will be described using animage processing apparatus as an example. However, the presentembodiment is also applicable to other devices that can perform thenumber recognition processing (character recognition processing) withoutbeing limited to an information processing server as a computer. Suchdevices may include, for instance, a personal computer, a digitalcamera, a mobile phone including a smartphone, a game machine, a tabletterminal, a monitoring camera system, or the like.

FIG. 1 illustrates a configuration example of an image processing systemincluding an image processing apparatus 100 according to the presentembodiment. The image processing system illustrated in FIG. 1 generallyincludes the image processing apparatus 100, an image capturingapparatus 112, a PLC 115, a database 116, and a network 111. In theimage processing system, the image capturing apparatus 112 generates animage obtained by capturing indication of a display 114, and the imageprocessing apparatus 10 performs processing such as number recognitionprocessing and output determination processing described below on acaptured image.

The image processing apparatus 100 includes respective componentsdescribed below such as a control unit 101, a display device 104, astorage device 105, and an input device 106. The control unit 101includes one or more processors such as a CPU, a ROM that is anon-volatile memory, and a RAM that is a volatile memory. The controlunit 101 controls all the image processing apparatus 100 by loading andexecuting, on the RAM, a program stored in the ROM. The program mayinclude an operation processing procedure of the control unit 101 (forinstance, a program of computer startup processing or basic input/outputprocessing), and a program for number recognition processing and outputdetermination processing described below. The RAM functions as a mainmemory of the processor and provides a work area when the CPU executesvarious types of processing.

The display device 104 includes, for instance, a liquid crystal or anorganic EL panel, and displays an application program executed undercontrol of the control unit 101, that is, an execution status of anapplication program, an image from a digital camera, or the like. Thestorage device 105 includes, for instance, a hard disk drive(hereinafter, HDD), a solid state drive (hereinafter, SSD) or the like,and saves an application program, data, a library, or the like. Theinput device 106 includes, for instance, a pointing device, a keyboardor the like, and accepts an operation instruction from a user of theimage processing apparatus 100.

A media drive 107 includes, for instance, a storage medium mounting unit(media drive), and enables reading out data of an image captured by theimage capturing apparatus 112 that is an external apparatus when aremovable storage medium (non-volatile memory device) is mounted on themedia drive 107.

A network interface 108 (also simply referred to as an I/F 108)transmits and receives data to and from a device that can communicatesuch as the image capturing apparatus 112, via a communication line 103and the network 111. For instance, the control unit 101 transmitsinformation of an image capture instruction to the image capturingapparatus 112 via the network interface 108, or receives a capturedimage from the image capturing apparatus 112.

A system bus 102 includes an address bus, a data bus, and a control busconnecting respective units of the image processing apparatus 100described above.

The image capturing apparatus 112 includes, for instance, a digitalcamera, and is connected to the network 111 via a communication path113. The image capturing apparatus 112 periodically captures an image ofthe display 114 described below, and generates gray scale image data,for instance. Then, the image capturing apparatus 112 transmits theimage data to the image processing apparatus 100 via the network 111.The image data received by the image processing apparatus 100 is savedin the storage device 105 as image data to be processed.

The display 114 includes a segment display to be measured, and displaysa predetermined character by controlling lighting of a plurality ofsegments. For instance, in the present embodiment, the segment displaythat displays physical quantity such as temperature, pressure, rotationspeed, or a distance measured by a measuring device (not illustrated) asa number by switching lighting and non-lighting of seven segments willbe described as an example.

The PLC (Programmable Logic Controller) 115 is a control device used tocontrol an operation of the image processing apparatus 100 incombination with an external device other than the image processingapparatus 100. In a case of executing an application program of theimage processing apparatus 100 in combination with the external device,the PLC 115 transmits information of an execution instruction to theimage processing apparatus 100. Additionally, the PLC 115 can store anexecution result of an application program of the image processingapparatus 100 in a register of the PLC 115, and the execution result canbe used for combination with another external device.

On the other hand, the control unit 101 of the image processingapparatus 100 can save an execution result of an application program ofthe image processing apparatus 100 in the PLC 115 or the database 116.In this case, the control unit 101 transmits the execution result fromthe network interface 108 to the PLC 115 or the database 116 via thenetwork 111.

The database 116 enables retrieving or registering data via a network,and for instance, receives an execution result of an application programof the image processing apparatus 100 and saves the execution result inthe database.

Note that, in the present embodiment, time required for switchingindication (for instance, a number) of the display 114 is sufficientlyshorter than exposure time of the image capturing apparatus 112. Thus,even when time required for switching indication extends across exposuretime for capturing two images, influence on the captured images isextremely small and there is no influence on number recognitionprocessing. Additionally, a sampling period of the measuring device inthe present embodiment will be described assuming that the samplingperiod is sufficiently longer than a frame rate of the image capturingapparatus 112, and measurement by the measuring device is not performedtwice or more during capture of three images in the present embodiment.

Aspects of Indication in Display

Next, an aspect of indication of the display 114 (segment display) willbe described by referring to FIG. 2. A character (here, a number)displayed by the display 114 is recognized by the image processingapparatus 100 by using a captured image.

A display unit 200 includes a segment including an LED or liquidcrystal, and displays, as a number, a measurement value of the measuringdevice that measures physical quantity such as temperature and pressuredescribed above. In the display unit 200, for instance, segments of LEDsor liquid crystal at seven positions are provided as a set, and a singlecharacter (here, number) is formed by lighting or non-lighting patternsof the set of segments. In the example illustrated in FIG. 2, a segment201 is a non-lighting part and a segment 202 is a lighting part. In theexample of the display unit 200 illustrated in FIG. 2, an example ofdisplaying a measurement value “123.4” by using four sets of segments isindicated.

Note that indication of the display 114 is not limited to this example,and, for instance, the display unit 200 may include a display panel suchas a liquid crystal panel. In this case, an image representing acharacter (here, a number) displayed by a set of segments may bedisplayed on the display panel of the display unit 200.

In a case where each segment of the display unit 200 includes an LED,the lighting part of a set of segments (lighting segments) is displayedbrighter than a background or the non-lighting part (that is,non-lighting segments). Additionally, in a case where each segment ofthe display unit 200 includes liquid crystal, the lighting part of a setof segments is displayed darker than a background or the non-lightingpart.

The display unit 200 may include a decimal point display unit 203 thatrepresents a decimal point. In the case of a display that can display adecimal point, the decimal point display unit is disposed at a lowerright position of the display unit that displays a numerical value. Thedecimal point display unit 203 exemplified in FIG. 2 indicates anexample in which the decimal point display unit is not lighting, and adecimal point display unit 204 indicates an example in which the decimalpoint display unit is lighting. In this way, in a case where ameasurement value to be displayed has a decimal point, the display unit200 represents a measurement value having a decimal point by switchinglighting and non-lighting of the decimal point display unit at anappropriate position.

Number Recognition Processing

Next, by referring to FIGS. 3 and 4, processing executed by the imageprocessing apparatus 100 of recognizing a character displayed by thedisplay 114 (the processing will also simply be referred to as numberrecognition processing) will be described.

In the number recognition processing, the image processing apparatus 100first extracts, from a captured image of the display 114, a region inwhich the display unit 200 is displayed. Then, the extracted region ofthe display unit 200 is further divided into a single-digit region(referred to as a single-character region), and the number recognitionprocessing is executed in the divided single-character region. FIG. 3illustrates an example of image data of a single-character region of acaptured image.

FIG. 3 illustrates seven segments 301 to 307 in a single-characterregion. Hereinafter, segments 301 to 307 are also referred to as segNo.1 to segNo. 7, respectively.

A point 308 is a point that bisects a top border constituting image dataof the single-character region. Additionally, a point 309 and a point310 are points that trisect a right border constituting image data ofthe single-character region. For instance, it is assumed that an upperpoint is the point 309 and a lower point is the point 310. A point 311is a point that bisects a bottom border constituting image data of thesingle-character region. Further, a point 312 and a point 313 are pointsthat trisect a left border of image data of the single-character region.It is assumed that an upper point is the point 313 and a lower point isthe point 312.

An intersecting point 314 is an intersecting point of a straight lineconnecting the point 308 and the point 311, and a straight lineconnecting the point 313 and the point 309. Additionally, anintersecting point 315 is an intersecting point of a straight lineconnecting the point 308 and the point 311 and a straight lineconnecting the point 312 and the point 310.

In the case of determining whether segNo. 1 (segment 301) is lighting ornot lighting, the control unit 101 sequentially reads pixel values fromthe point 309 to the intersecting point 314, and determines whetherthere exists a pixel having a luminance equal to or greater than aconstant value. In a case where the control unit 101 determines thatthere exists a pixel having a luminance equal to or greater than aconstant value, it is assumed that segNo. 1 is lighting and adetermination result related to a lighting state of the segment isstored in the RAM in association with segNo. 1. On the other hand, in acase where the control unit 101 determines that there exists no pixelhaving a luminance equal to or greater than a constant value, it isassumed that segNo. 1 is not lighting and a determination result isstored in the RAM in association with segNo. 1.

Note that the above-described lighting determination method is adetermination method applicable in a case where the display 114 such asan LED is a segment having a higher luminance in a state where thedisplay 114 is lighting than in the non-lighting state. On the otherhand, in a case where the display 114 includes. for instance, liquidcrystal, a different method may be used as the method for determiningwhether the segment is lighting or not lighting. For instance, thecontrol unit 101 sequentially reads pixel values from the point 309 tothe intersecting point 314, and determines whether there exists a pixelhaving a luminance equal to or less than a constant value. Then, in acase where the control unit 101 determines that there exists a pixelhaving a luminance equal to or less than a constant value, it is assumedthat segNo. 1 (segment 301) is lighting, and a determination result isstored in the RAM in association with segNo. 1.

Similarly, as for segNo. 2 (segment 302), the control unit 101sequentially reads pixel values from the intersecting point 314 to theintersecting point 315, and determines whether or not the segment islighting by the above-described method. Then, a determination result isstored in the RAM in association with segNo. 2. Similarly, the controlunit 101 sequentially reads pixel values from the point 310 to theintersecting point 315 as for segNo. 3, and sequentially reads pixelvalues from the point 311 to the intersecting point 315 as for segNo. 4,and performs similar processing. Further, the control unit 101sequentially reads pixel values from the point 312 to the intersectingpoint 315 as for segNo. 5, sequentially reads pixel values from thepoint 313 to the intersecting point 314 as for segNo. 6, andsequentially reads pixel values from the point 308 to the intersectingpoint 314 as for segNo. 7, and performs similar processing.

FIG. 4 illustrates a relation between a determination result related toa lighting state of each segment and a number displayed.

A management number column 401 is a column in which a management numberfor uniquely specifying data in a table is described. A lighting pattern402 is a column in which a pattern in a lighting state is indicated.Columns 403 to 409 are columns in which lighting conditions ofcorresponding segNo. 1 to segNo. 7 are indicated, respectively. A column410 is a column in which an output value to be determined isrepresented.

Additionally, a row 411 to a row 426 corresponding to specificmanagement numbers have respective output values (numerical values)associated with lighting conditions of corresponding segments. Forinstance, in a case where segNo. 1 to segNo. 7 are all ON, lightingstates of the segments match a pattern indicated in the row 422. Thus,in the lighting states, an output value to be output becomes 8.Additionally, for instance, in a case where only segNo. 2 is ON and allthe other segments are OFF, lighting states of the segments match therow 425. Thus, in the lighting states, an output value to be outputbecomes “−” (minus).

In this way, the control unit 101 holds, in the RAM, a determinationresult of the lighting state of each segment obtained by using thedetermination method illustrated in FIG. 3. Then, the control unit 101uses the table of FIG. 4 to search for a row indicating a lighting statethat matches the determination result of the lighting state. At thistime, in a case where there exists a row that matches the determinationresult of the lighting state, a value (output value) of the column 410of the matching row is held in the RAM as a number recognition result.On the other hand, in a case where a row that matches the determinationresult is not found, the control unit 101 holds the determination resultin the RAM as recognition failure. The control unit 101 performs similarprocessing on each of numbers displayed by the display 114, and holds arecognized number in the RAM.

Relation Between Change of Indication of Display and Imaging Timing

Next, by referring to FIG. 5, a relation between a change of indicationof the display 114 and imaging timing by the image capturing apparatus112 will be described.

As described above, a measurement value displayed by the display 114 isa measurement result of measurement performed by a measurement devicefor each sampling period. In the example illustrated in FIG. 5, atemporal change related to timing at which a measurement value displayedby the display 114 changes from “2” to “5.”

A time axis 500 represents a temporal change, and each of t1 to t6indicates timing at which the image capturing apparatus 112 startsexposure. Indication 501 represents a measurement value displayed by thedisplay 114. The example illustrated in FIG. 5 indicates that ameasurement value displayed at time t1 to t4 is “2” (indication 50 a).Additionally, it is indicated that a measurement value displayed at timet5 to t6 is “5” (501 b). At this time, a measurement value displayedchanges from “2” (indication 501 a) to “5” (501 b) between t4 and t5.

A captured image 502 is an image obtained by the image capturingapparatus 112 that has started exposure and captured the display 114 atthe timing t1 to t6. In the present embodiment, an image to be capturedis a gray scale image, and the gray scale image is generally classifiedinto the following three types of regions. The first type is a region(region 503) where a state of a segment of the display 114 is thenon-lighting state. The second type is a region (region 504) where astate of a segment of the display 114 is the lighting state. Then, thethird type is a region (region 505) where a state of a segment changesduring exposure, and is a region having a luminance of an intermediatestate between the lighting state and the non-lighting state.

Ina case where exposure is started at the timing t1 to 3, a capturedimage 502 a in which the display 114 displays a measurement value “2” isobtained. In a case where exposure is started at the timing t5 to t6, acaptured image 502 b in which the display 114 displays a measurementvalue “5” is obtained. On the other hand, in a case where exposure isstarted at the timing t4, a measurement value changes from “2” to “5”while the image capturing apparatus 112 is performing exposure. Thus, acaptured image 502 c becomes an image in which a changed segmentincludes a luminance of an intermediate state between the lighting stateand the non-lighting state.

In the case of the captured image 502 a or 502 b, the control unit 101can detect a measurement value “2” in the image 502 a by comparing withthe table illustrated in FIG. 4 according to the determination method ofthe lighting state illustrated in FIG. 3. On the other hand, ameasurement value “5” can be obtained in the image 502 b. In contrast,in a case where a captured image is the captured image 502 c captured attiming t4, the lighting state cannot be obtained accurately. In thiscase, there is a possibility that a number recognition result that isnot applied to a number in the table such as the row 426 illustrated inFIG. 4 is output, or a number recognition result “8” that is differentfrom the measurement value displayed by the display 114 is output. Inthe following description, processing (output determination processing)that enables outputting of an appropriate number (character) as anoutput value even in a case where a captured image such as the capturedimage 502 c is present will be described.

Series of Operations Related to Output Determination Processing UsingTime Series Images

Next, a series of operations of output determination processing usingtime series images according to the present embodiment will bedescribed, by referring to FIG. 6. The output determination processingis processing in which a recognition result (number in this example) ofeach of time series captured images is used to determine whichrecognition result (number) among a plurality of recognition results isto be output. A series of operations related to the output determinationprocessing are realized by the control unit 101 that loads and executes,on the RAM, a number recognition application program stored in the ROM.Additionally, initial setting of the image processing apparatus 100 iscompleted after activation and the image processing apparatus 100 is inan execution waiting state of the processing, and starts by receiving astart operation from the input device 106 or via the network. Theinitial setting includes setting of PTZ (pan/tilt/zoom) of the imagecapturing apparatus 112 to keep the display 114 within a field of view.Additionally, the initial setting includes appropriately setting anumber recognition processing parameter for the number recognitionapplication program, such as a luminance threshold value for determiningthe lighting state of the segment described by referring to FIG. 3.

At step S601, the control unit 101 initializes a value of a variable ifor a loop counter of camera image capture to zero.

At step S602, the control unit 101 transmits an image capture startcommand to the image capturing apparatus 112 via the network interface108 by executing the number recognition application program.Accordingly, the image capturing apparatus 112 starts image capture ofthe display 114, and transmits an obtained captured image to the imageprocessing apparatus 100 via the network 111.

The number recognition application program of the image processingapparatus 100 stores the captured image received via the networkinterface 108 in the RAM in association with the value of the loopcounter i.

At step S603, the control unit 101 executes the number recognitionprocessing to recognize a number displayed by the display 114 from thecaptured image stored at step S602. Specifically, the control unit 101uses the method described by referring to FIG. 3.

The control unit 101 stores number information of the display 114obtained by the number recognition processing in the RAM in associationwith the value of the loop counter i.

At step S604, the control unit 101 determines the number of loops of theloop counter i stored in the RAM. Here, N is a constant of the maximumnumber of loops, and in the present embodiment, the number of loops is,for instance, three. In a case where the loop counter i is less than theconstant N, the control unit 101 returns the processing to step S602 andrepeats the number recognition processing. In a case where the loopcounter i is equal to or greater than the constant N, the processingexits the loop and proceeds to the determination processing step of arecognized number.

At step S605, the control unit 101 increments the value of the loopcounter i stored in the RAM. The value of the loop counter i is changed,and thus duplication of the captured images acquired at step S602 andduplication of the numbers recognized at step S603 are avoided.

The determination processing steps S606 to S610 are processing in whicha number to be output as indication of the display 114 is determined byusing the three images captured at step S602 (that is, determine whichnumber among the recognition results of the three images is to beoutput). Here, FIGS. 7A to 7D illustrate images of the results of imagecapture with respect to a change of numbers of the display 114. That is,since N at step S604 is three, there are three captured images capturedat step S602.

FIG. 7A illustrates three captured images captured at the timing t1 tot3 illustrated in FIG. 5. During the time t1 and t3, a number of ameasurement value displayed by the display 114 does not change. Thus,all the images captured are of an identical number.

FIG. 7B illustrates three captured images captured at the timing t2 tot4 illustrated in FIG. 5. In the image exposed at the timing t4, anumber of a measurement value displayed by the display 114 changesduring exposure. Thus, although in two of the three images, a number ofa measurement value displayed by the display 114 is captured, but thelast one image becomes an image in which an accurate measurement valueis not captured due to a change of the display 114 during exposure.

FIG. 7C illustrates three captured images captured at the timing t3 tot5 of FIG. 5. In the image exposed at the timing t4, a number of ameasurement value displayed by the display 114 changes during exposure.Thus, the first image of the three images is a captured image in which anumber obtained before the number of the measurement value displayed bythe display 114 has changed is captured. The second image becomes acaptured image in which an accurate measurement value is not captureddue to a change of the display 114 during exposure, and the third imagebecomes a captured image in which a number obtained after the number ofthe measurement value displayed by the display 114 has changed iscaptured.

FIG. 7D illustrates three images captured at the timing t4 to 16 of FIG.5. The image exposed at the timing t4 is the first image in which anaccurate measurement value is not captured due to a change of a numberof a measurement value displayed by the display 114 during exposure. Thesubsequent two (second and third) images are captured images in which anumber of a measurement value displayed by the display 114 is captured.Here, as described above by referring to FIG. 1, switching time ofnumbers of the display 114 in the present embodiment is sufficientlyshorter than exposure time of the image capturing apparatus 112. Thus,the three images captured by the image capturing apparatus 112 match anyone of the four patterns illustrated in FIGS. 7A to 7D (patterns (A) to(D), respectively).

Then, in the determination processing of S606 to S610, the control unit101 determines a measurement value to be output as indication of thedisplay 114 by determining which of the patterns (A) to (D) the threeimages captured in time series match.

At step S606, the control unit 101 determines whether or not all theresults of the numbers recognized at step S603 are identical. In a casewhere all the recognized numbers are an identical number, the controlunit 101 proceeds to step S607. This case corresponds to the patternillustrated in FIG. 7A, and all the number recognition results of thethree images are identical. Thus, at step S607, the control unit 101determines that the display 114 has not changed during exposure of theimage capturing apparatus 112, and holds, in the RAM, arbitraryrecognition result (or constantly the last recognition result) of thethree images. On the other hand, in a case where the control unit 101determines at step S606 that the recognition results are differentnumbers, the control unit 101 proceeds to step S608.

At step S608, the control unit 101 determines whether or not the numbersrecognized at S603 are of two types. In a case where the recognizednumbers are of two types, the control unit 101 proceeds to step S609.This case corresponds to the pattern illustrated in FIG. 7B or 7D. Thatis, since the number recognition results are of two types, it can bedetermined that the number recognition result of one image is a numberrecognition result obtained in exposure during a change of the display114. On the other hand, two images in which the number recognitionresults are identical can be determined as a number recognition resultobtained not during the change. Then, at step S609, the control unit 101holds, in the RAM, a number recognized from the largest number of images(two, in this case).

On the other hand, in a case where the control unit 101 determines atstep S608 that the number recognition result at step S603 is not of twotypes (that is, three types), the control unit 101 proceeds to stepS610. This case corresponds to FIG. 7C. Since the number recognitionresults are of three types, the control unit 101 can determine that thesecond number recognition result is a number recognition result obtainedin exposure during a change of the display 114. Additionally, it can bedetermined that the first image is the number recognition resultobtained before the display 114 changes, and the third image is thenumber recognition result obtained after the change. At step S610, thecontrol unit 101 holds the last recognized number. Note that, at stepS610, the control unit 101 may hold the first recognized number.

At step S611, the control unit 101 outputs the number recognition resultheld in the RAM at the immediately preceding step (that is. S607, S609,S610) to an external system as an output determined by the imageprocessing apparatus 100. The control unit 101 may store the determinedoutput as a binary expression of numerical data or as text data in thestorage device 105 or the like in the image processing apparatus 100.Additionally, the control unit 101 may save the determined output in astorage medium in the media drive 107, or record the determined outputin an external storage medium (such as a database) connected to thenetwork 111. The control unit 101 subsequently ends a series ofoperations related to the output determination processing.

As described above, in the present embodiment, a character (number) tobe output is determined based upon a change aspect in time series ofnumbers recognized from three captured images of a single character(number) captured in time series. At this time, assuming that thecaptured images include the captured image exposed during a change ofthe display 114, it is considered that one erroneous number isrecognized in a case where there exist three number recognition results.In a case where two types of numbers are recognized as numberrecognition results, an identical number recognized from a plurality ofimages is employed, excluding the number recognized from one image. Onthe other hand, in a case where three numbers are recognized as numberrecognition results, an erroneous number obtained in the course of achange appears in the middle image, and thus a numerical value of theimage captured last (or first) is employed. In this way, even in a casewhere an erroneous character due to a change of indication of thedisplay is recognized, an accurate recognition result can be finallyoutput.

Note that in the present embodiment described above, a seven-segmentdisplay including seven segments is described and used as an example.However, the present embodiment is also applicable to a 14-segmentdisplay including 14 segments, or a 16-segment display including 16segments. In such a case, the determination processing and the tabledata of the lighting state of each segment may be changed in accordancewith the number of segments.

Additionally, in the present embodiment described above, the case wherea character displayed by the display 114 is a number is described andused as an example. However, the above-described processing can beapplied to a character such an alphabet and a symbol.

Further, in the present embodiment described above, the case where N atstep S604 (that is, the number of images captured in time series) isthree is described and used as an example. However, the outputdetermination processing illustrated in FIG. 6 may be performed with Nbeing four or more (the number of images to be captured being four ormore). However, it is assumed that measurement by the measuring deviceis not performed two or more times during capture of N images in thepresent embodiment.

Further, in the above-described embodiment, the case of separatingnumbers of the display 114 into single digits and executing the numberrecognition processing for each digit. However, a number recognitionprocessing method in which numbers displayed by a display in an imageare recognized at one time may be employed, instead of performing thenumber recognition processing for each digit.

Additionally, in the above-described embodiment, the method ofrecognizing a number based upon a so-called rule-based algorithm thatapplies the lighting state and the non-lighting state of a segment tothe pattern illustrated in FIG. 4. However, the number recognitionprocessing may not necessarily be recognized by the rule-basedalgorithm. For instance, a method in which an image of the display 114is learned in advance by using machine learning, and a displayed number(character) is recognized from a captured image of the display 114 maybe used. As the method that uses machine learning, for instance, amethod in which a number (character) is recognized by using a deepneural network including a convolutional layer (for instance,Convolutional Neural Network) may be used. In this way, a number(character) can be recognized based upon a pattern of the lighting stateand the non-lighting state of a segment, without manually providing anamount of characteristics or algorithm in advance.

In the above-described example, the case of recognizing a maximum ofthree numbers from three captured images is described and used as anexample. However, similar processing can also be applied to the casewhere a result of an unrecognizable number is obtained. That is, when arecognition result of an unrecognizable number is obtained, the resultmay be treated equally to the case where one type of character isrecognized.

Second Embodiment

Next, a second embodiment will be described. Although the secondembodiment differs from the first embodiment in terms of a part of theoperation of the number determination processing, other components orprocessing are similar to those of the first embodiment. That is,components of an image processing system and number recognitionprocessing with respect to indication of a display 114 are similar tothose of the first embodiment. Thus, identical or substantiallyidentical components will be denoted by identical reference signs, anddescription of these components will be omitted. Description will bemade focusing on different components.

Series of Operations Related to Output Determination Processing UsingTime Series Images

A series of operations related to output determination processing usingtime series images according to the present embodiment will be describedby referring to FIG. 8.

At step S801, a control unit 101 transmits an image capture startcommand to an image capturing apparatus 112, in a similar manner to stepS602 described above. In response to this, the image capturing apparatus112 captures indication of the display 114, and transmits obtained imagedata to an image processing apparatus 100 via a network 111. The controlunit 101 stores, in a RAM, the image data received from the imagecapturing apparatus 112.

At step S802, the control unit 101 executes number recognitionprocessing for recognizing a number displayed by the display 114 fromthe image data stored at S801. At this time, the control unit 101executes the number recognition processing described above in the firstembodiment. Then, at step S803, the control unit 101 holds the numberrecognized at S802 in a variable R1 reserved in the RAM.

Next, at step S804, the control unit 101 stores image data captured bythe image capturing apparatus 112 in a similar manner to S801, andfurther, the control unit 101 executes at step S805 the numberrecognition processing on the stored image data in a similar manner toS802. Then, at step S806, the control unit 101 holds the numberrecognized at S805 in a variable R2 reserved in the RAM.

That is, at steps S801 to S806, processing of holding the resultobtained by performing the number recognition processing on a precedingimage of successive time series images, and the result obtained byperforming the number recognition processing on a subsequent image inthe variables R1 and R2, respectively.

At step S807, the control unit 101 executes determination processing inwhich R1 and R2 that are the number recognition results held at S803 andS806 are compared. In a case where the control unit 101 determines thatR1 and R2 do not match, the control unit 101 determines that indicationof the display 114 has changed during execution of the numberrecognition processing, and proceeds the processing to step S808. Atstep S808, the control unit 101 substitutes R2 that is a result ofsubsequent recognition into the variable of R1. The control unit 101proceeds to S804 after completion of the processing at step S808, andrepeats the processing until recognition results match in thedetermination at S807. In this way, a correct number can be output anddetermined when an identical recognition result is successively obtainedwhile dynamically changing the number of a plurality of time seriescaptured images.

On the other hand, in a case where the control unit 101 determines thatR1 and R2 match at step S807, the control unit 101 proceeds theprocessing to step S809. At step S809, the control unit 101 determinesthat the recognition result R1 is a number displayed by the display 114,and outputs the recognition result R1 to an external system. It isneedless to say that the result to be output may be the result held inR2.

Note that the control unit 101 may store the determination result as abinary expression of numerical data, or as text data in a storage device105 or the like in the image processing apparatus 100. Additionally, thedetermination result may be stored in a storage medium in a media drive107, or may be recorded in an external storage medium (such as adatabase) connected to the network 111.

As described above, similarly, according to the present embodiment, adisplayed character (number) is determined based upon successivecaptured images of a single character (number) captured in time series.At this time, assuming that the captured images may include the capturedimage exposed during a change of the display 114, it is considered thatan erroneous number is recognized in a case where successive numberrecognition results do not match each other. On the contrary, in a casewhere successive number recognition results match each other, it isdetermined that there is a correct number recognition result without anyerroneously recognized number and the number recognition result isemployed as an output. In this way, even in a case where an erroneouscharacter due to a change of indication of a display is recognized, afinally accurate recognition result can be output.

Other Embodiments

Embodiment(s) of the present invention can also be realized by acomputer of a system or apparatus that reads out and executes computerexecutable instructions (e.g., one or more programs) recorded on astorage medium (which may also be referred to more fully as a‘non-transitory computer-readable storage medium’) to perform thefunctions of one or more of the above-described embodiment(s) and/orthat includes one or more circuits (e.g., application specificintegrated circuit (ASIC)) for performing the functions of one or moreof the above-described embodiment(s), and by a method performed by thecomputer of the system or apparatus by, for example, reading out andexecuting the computer executable instructions from the storage mediumto perform the functions of one or more of the above-describedembodiment(s) and/or controlling the one or more circuits to perform thefunctions of one or more of the above-described embodiment(s). Thecomputer may comprise one or more processors (e.g., central processingunit (CPU), micro processing unit (MPU)) and may include a network ofseparate computers or separate processors to read out and execute thecomputer executable instructions. The computer executable instructionsmay be provided to the computer, for example, from a network or thestorage medium. The storage medium may include, for example, one or moreof a hard disk, a random-access memory (RAM), a read only memory (ROM),a storage of distributed computing systems, an optical disk (such as acompact disc (CD), digital versatile disc (DVD), or Blu-ray Disc (BD)™),a flash memory device, a memory card, and the like.

While the present invention has been described with reference toexemplary embodiments, it is to be understood that the invention is notlimited to the disclosed exemplary embodiments. The scope of thefollowing claims is to be accorded the broadest interpretation so as toencompass all such modifications and equivalent structures andfunctions.

This application claims the benefit of Japanese Patent Application No.2019-239276, filed Dec. 27, 2019 which is hereby incorporated byreference herein in its entirety.

What is claimed is:
 1. An image processing apparatus comprising: one ormore processors; and a memory storing instructions which, when theinstructions are executed by the one or more processors, cause the imageprocessing apparatus to function as: a recognition unit configured toacquire a plurality of captured images of characters captured in timeseries, each of the characters including a plurality of segments andthat changes in time series, and recognize the characters captured foreach of the plurality of captured images; and a determination unitconfigured to determine which one of the characters recognized from eachof the plurality of captured images is to be output, wherein thedetermination unit determines, in accordance with which one of a firstaspect, a second aspect and a third aspect a change in time series ofthe characters recognized from each of the plurality of captured imagesis, which one of the characters recognized from each of the plurality ofcaptured images is to be output.
 2. The image processing apparatusaccording to claim 1, wherein, in a case where the characters recognizedfrom each of the plurality of captured images are three types ofcharacters, which corresponds to the first aspect, the determinationunit determines, among the characters recognized from each of theplurality of captured images, a character recognized from a lastcaptured image as a character to be output.
 3. The image processingapparatus according to claim 1, wherein, in a case where the charactersrecognized from each of the plurality of captured images are two typesof characters, which corresponds to the second aspect, the determinationunit determines, among the characters recognized from each of theplurality of captured images, a character recognized from the largestnumber of images as a character to be output.
 4. The image processingapparatus according to claim 1, wherein, in a case where the charactersrecognized from each of the plurality of captured images are anidentical character, which corresponds to the third aspect, thedetermination unit determines the recognized character as a character tobe output.
 5. The image processing apparatus according to claim 1,wherein the single character includes 7, 14, or 16 segments and isdisplayed by a display.
 6. The image processing apparatus according toclaim 1, wherein the recognition unit recognizes the single character inaccordance with patterns of a lighting state and a non-lighting state ofthe plurality of segments.
 7. An image processing system comprising: animage capturing unit configured to generate a plurality of capturedimages of characters captured in time series, each of the charactersincluding a plurality of segments; and an image processing apparatuscomprising: one or more processors; and a memory storing instructionswhich, when the instructions are executed by the one or more processors,cause the image processing apparatus to function as: a recognition unitconfigured to acquire a plurality of captured images of characterscaptured in time series, each of the characters including a plurality ofsegments and that changes in time series, and recognize the characterscaptured for each of the plurality of captured images; and adetermination unit configured to determine which one of the charactersrecognized from each of the plurality of captured images is to beoutput, wherein the determination unit determines, in accordance withwhich one of a first aspect, a second aspect and a third aspect a changein time series of the characters recognized from each of the pluralityof captured images is, which one of the characters recognized from eachof the plurality of captured images is to be output.
 8. A control methodof an image processing apparatus, the control method comprising:acquiring a plurality of captured images of characters captured in timeseries, each of the characters including a plurality of segments andthat changes in time series, and recognizing the characters captured foreach of the plurality of captured images; and determining which one ofthe characters recognized from each of the plurality of captured imagesis to be output, wherein the determining includes determining, inaccordance with which one of a first aspect, a second aspect and a thirdaspect a change in time series of the characters recognized from each ofthe plurality of captured images is, which one of the charactersrecognized from each of the plurality of captured images is to beoutput.
 9. A non-transitory computer-readable storage medium comprisinginstructions for performing the control method of an image processingapparatus, the control method comprising: acquiring a plurality ofcaptured images of characters captured in time series, each of thecharacters including a plurality of segments and that changes in timeseries, and recognizing the characters captured for each of theplurality of captured images; and determining which one of thecharacters recognized from each of the plurality of captured images isto be output, wherein the determining includes determining, inaccordance with which one of a first aspect, a second aspect and a thirdaspect a change in time series of the characters recognized from each ofthe plurality of captured images is, which one of the charactersrecognized from each of the plurality of captured images is to beoutput.